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SYNOPSIS 


Todays mega -proj ects transcend the traditional trajectories 
traced within national and technological limitations. Powers 
unleashed by internationalization of initiatives, in for example 
space exploration and environmental protection, are arguably only 
temporarily suppressed by narrower national, economic and 
professional disagreements as to how best they should be harnessed. 

While the world gets its act together there is time to 
develop the technologies of such supra-mega-proj ect management 
that will synergise truly diverse resources and smoothly mesh 
their interfaces. Such mega-projects and their management need 
to be realistically evaluated, when implementing such improvements. 

This paper examines current approaches to evaluating mega- 
projects and questions the validity of extrapolations to the supra 
-mega-projects of the future. Alternatives to improve such 
evaluations are proposed and described. 


1. MEGAPRQJECTS OF TODAY AND TOMORROW 


1.1 From projects to mega- p roj ect s 

First focusing on projects, most simplified definitions 
portray a project as an amalgamation of inter-related non-routine 
non-recurrent activities aimed at specific objective(s) and with 
definite start and finish points in time. Distinctions between 
project and general management stress the limitation of resources, 
specificity of goal(s) and the short sharp imperatives facing 
temporary project teams as against the longer term almost self- 
perpetuating and relatively routine roles of general managers 
maintaining streamlined operations in steady state scenarios. 

To distinguish mega - proj ects from projects is less simple; 
and admittedly a matter of choice: whether to use criteria based 
on money value/ cost; or on impact to the community and/or 
environment; or on the number of people deployed; or key 
parameters involved such as hectares of land, cubic meters of 
earth or concrete, kilometers of road or tunnel; or a combination 
of the foregoing. 

1.2 Examples of mega-projects 


Traces of past megaprojects continue to fascinate us as for 



example in the great pyramids of Egypt, the Great Wall of China, 
mega- irrigation works and other edifices of ancient civilisations. 

Voyages of discovery were megaprojects whether across the 
unchartered seas, into unexplored forests of yore, into the depths 
of the Ocean or into Space. What springs to mind today may be the 
mega-undertakings such as the Space Station, Moon Base or Mission 
to Mars. It is useful to consider these in the context of some 
contemporary or imminent examples on Earth itself; each of which 
had/ has/ will have a mega- impact on its people and environment: 

(a) the proposed Three Gorges Project is on the 6300 km. long 
Yangtze river which has a drainage area of 1.3 million sq. km 
and a 'mean annual runoff of 1 trillion cu nr . The proposed 
concrete gravity dam requiring 15.27 million cu m. of concrete 
will be 2546 m long and 175 m high at the crest. The reservoir 
capacity will be 39.3 billion cu m. and the two hydropower plants 
will have a total installed capacity of 17,680 MW. 

On the other hand 23,793 hectares of farmland, 13 towns, 
140 townships and 1500 villages will be affected by the reservoir 
and the population to be moved may well be over 1.1 million. 

The cost was estimated at 57 billion yuan (about US$ 10 
billion) based on 1990 prices, the net economic benefit at 13.2 
billion yuan, the Economic Rate of return at 14.5%, the Financial 
Rate of Return at 11%; and the Payback Period at 20.6 years which 
Is about 2 1/2 years after estimated total completion. 

(b) The Channel Tunnel will soon change the character of earth- 
bound traffic between the UK and the rest of Europe. However the 
approximately US$ 13 Billion project is already 60% over budget 
and over one year behind schedule. 

Furthermore the US$ 1.8 billion claim by the consortium of 
contractors recently resulted in a deadlock in Paris which 
perhaps exemplifies that the problems of megaprojects are not very 
different from smaller ones except that the repercussions can be 
that much greater. The potential for problems is perhaps 
heightened by the interactions between the many powerful 
personalities of the multi- project participants; but then the 
opportunities for bigger and better solutions is also hopefully 
heightened on synergistic grounds. 

(c) The proposed new airport in Hong Kong vividly illustrates 
the problems of multi - stake -holder interests in mega-projects. The 
program estimated at approximately US$20 Billion, has been split 
Into 10 'Core' projects. Some of these are underway; while others 
may be modified in order to meet the self-imposed deadline of 
1997; and also as a compromise in the on-going prestige battle 
between Britain and China on broader issues. 

This megaproject includes transportation links to the 
airport that is being built mostly on land reclaimed from the sea. 
The 'Core' projects include long span bridges and road and rail 



links and interchanges which are 'big' projects in themselves. 

(d) It is worth comparing the foregoing with the $ 2.7 billion 
new Denver International Airport megaproject In USA. At its 
ultimate capacity it will exceed the combined capacities of the 
worlds two largest airports at present 3 . 

(e) The Mahaweli Irrigation and hydropower program in Sri Lanka 
launched in the late 1970's sought to compress into the life span 
of one parliament, a previously phased out program that would 
have spanned 30 years and cost over US $ 1 billion. Foreign funds 
from many countries were injected to fuel this acceleration. It 
was not difficult to achieve (on paper) the required rates of 
return, by for example changing some designated crop areas from 
paddy to higher return yielding cash crops. The relative merits 
of the nature and magnitude of the foreign inputs are still being 
questioned by those who point out to the stifling of the domestic 
construction industry, for example. 

Even the foregoing small sample of contemporary mega- 
projects indicates the nature of the emerging agendas to be 
addressed; for example of the increasing size, scope and linkages 
of mega- proj ec ts ; and of the delicate interface management needed 
between the many powerful and sometimes culturally and 
technologically diverse project participants. Issues to be 
resolved sometimes include those arising from divergent value 
systems and from real or imagined impacts on the environment. 


1.3 The need for evaluation 

The foregoing random scan of a few mega - proj ec ts , 
illustrates the need for evaluating every project against relevant 
criteria. The stakes are so high and the variables so many that 
ground-rules need to be established, targets set and evaluated 
against from the outset for many reasons; including the efficient 
allocation of scarce resources, for example. 

President Clinton recently directed NASA to redesign Space 
Station 'Freedom' to make it more efficient and effective and 
capable of producing greater returns on investment. No doubt NASA 
has translated these into a set of detailed ' evaluatable ' targets. 
The imperatives for proper evaluation are just that much more 
compelling on mega-projects which can have mega- impact on so many 
living things; and even those who are still unborn. 

2 . EVALUATION 


2,1 What is evaluation? 

Unfortunately the word evaluation has been used in various 
contexts and with different connotations. It can for example mean 
different things to economists, engineers or human resource 
managers. This paper which espouses a multi-disciplinary approach 
to evaluation, takes evaluation to cover both ongoing and 
completed project reviews. It excludes pre-project appraisals of 



feasibility but must necessarily relate to the setting and 
validation of objectives at that stage. 

In comparison, Imboden defined^ evaluation as the ex-post 
analysis of an executed project' and project appraisal as the 'ex- 
ante analysis of a proposed project' . The Overseas Development 
Administration of UK defines it similarly^ 5 ; while using 
'monitoring' for reviews of ongoing projects. Here mid-project 
evaluation is taken to include a more detailed performance 
analysis against targets, than may be implied by monitoring. On 
the other hand Corrie extends tlje scope of evaluation to include 
planning and feasibility studies 0 . 

This paper also distinguishes evaluations from audits which 
may merely test compliance with management controls and 
regulations. Evaluation implies testing performance against pre- 
determined targets; that may or may not be adjusted for changed 
circumstances. More fundamentally, evaluation refers to the 
process of determining the merits, worth or value of things, or to 
the result of that process . It is implied that the reference is 
to the net value; so that both value and cost aspects are 
assessed . 

2.2 Whv evaluate? 

Briefly, project evaluators may seek (a) to assess 
performance on on-going or completed projects; in order to reward 
or reprimand participants; (b) to improve future performance (by 
lessons learnt from failures); (c) to improve future target 
setting (eg: incorporating weightings for project circumstances so 
as to yield more realistic targets). 

Top management, shareholders, governments or the community 
itself (in the case of mega-projects) would also want to know what 
went wrong and why, how bad it really was and how things could be 
improved. For example they may want to know the exact impact on 
project success of for instance the mirror defect in the Hubble 
Space Telescope or the jammed high gain antenna in the Galileo 
mission. 

While there are already many stake -holders in a smaller 
project, those with Interests in a mega-project multiply 
tremendously, eg: those affected by the environmental impact or 
whose tax payments may have contributed to the funding. 

2.3 How to evaluate? 

Will a systematised approach to evaluation of mega-projects 
provide the necessary answers? What of the grey areas that may 
need qualitative judgements? How does one evaluate the ability of 
the project team to hit the 'moving targets' that often result 
from changing project priorities and circumstances eg: sudden 
price restrictions; mid- stream scope expansions; or intermediate 
time targets, for sectional completion or meeting other 
milestones . 



Except in major disasters like the 'Challenger' or a breached 
dam, it is usually the time and cost targets that are over -run, as 
in the case of the Sydney Opera House; since the performance 
specifications can often be met by incurring more time and money. 
It is difficult to be excellent, fast and economical all at once, 
but that is the essence of a good project and of it's management. 


ZJ± Differentiating project success from that of its management 

Drastic changes in conditions affecting a project can result 
in project failure; and successful management can sometimes only 
mitigate the efforts of such failures. At the other extreme 
unrealistically 'easy' targets and very favourable conditions can 
precipitate project success in spite of management mishaps. 

The evaluator (s) should therefore be clear what/ who is 
being evaluated; whether it is the project, its management or 
both. Effectiveness (in achieving results) must be distinguished 
from efficiency (in optimising resource usage in achieving such 
results). 

The consequent distinction between 'impact' or 'outcome' 
evaluation (of results) and 'implementation' evaluation of process 
(and management) is self - explanatory. 

The formulation of suitable targets against which to measure 
success must therefore take account of such distinctions. 


3. CRITERIA AND INDICATORS OF SUCCESS 
3.1 Establishing criteri a of success 

Since every project is unique and the major stake -holders 
may have special priorities, it is essential to jointly establish 
the criteria by which success will ultimately be judged. Securing 
the agreement of multiple stake holders of a mega-project may well 
prove a formidable project by itself. Incorporating such agreed 
criteria in the project brief is therefore a primary management 
task. 


Criteria of project success which were traditionally based 
on the cost-time-quality tripod, have grown in both number and 
sophis tication . Health, safety and environmental criteria, as well 
as stake-holder satisfaction criteria are also often (consciously 
or otherwise) considered when evaluating the success level of a 
project or its management. For example Ashley et al listed in 1987 
six criteria most frequently used to measure construction project 
success as: budget performance; schedule performance; 
functionality; client satisfaction; contractor satisfaction and 
project manager/ team satisfaction 8 . 

As for increased sophistication; the cost criterion for 
instance may not be confined to initial capital cost; but may well 



include life-cycle cost; elemental (sectional) costs; cash flow 
profile factors etc for management or implementation evaluation 
and economic rates of return incorporating social benefits as 
well, for impact or outcome evaluation. Thus life-cycle cost, 
elemental costs etc. may be sub-criteria within the overall cost 
criterion. Return on capital is an example of a criterion relating 
to the overall project success itself. 

The relative significance of each criterion (and sub- 
criterion) would necessarily vary with the project priorities on 
different projects. Cost may be paramount in some, while timing or 
prestige / performance levels may be of the essence on others. 

The multiple criteria against which projects are to be 
evaluated often result in varying degrees of success/ failure 
against each criterion and sub-criterion. Figure 1 illustrates a 
model proposed by the author to illustrate a profile of 
performance against each such criterion (a,b, c....) and sub- 
criterion (al, a2, ...bl, b2, ... etc) . 

Appropriate measures to be used for each such criterion and 
sub-criterion need to be defined and suitable scales establishe^ 
for such measurement. Measurement can be by suitable 'indicators' 

3.2 Formulating indicators of success 

Indicators are needed here: (a) as proxies for measurement 
where direct measurement is difficult; (b) as short-hand symbols 
to measure vast quantities of data; (c) as short- cuts to a quick 
first approximation of the status of a project or its management; 
(d) to present such status through measurements against relevant 
criteria for example on the above-mentioned project performance 
profiles . 

Examples of such basic primary indicators can be cost/kg of 
payload launched or cost/ m 3 of usable space, In a Space mission; 
or cost/m of building, cost/ m of water stored or cost/MW of 
power generated etc. A combination of primary indicators may be 
designed to represent one criterion if necessary ie if one 
indicator Is deemed inadequate by itself. Each primary indicator 
in turn can be analysed into secondary (and tertiary) indicators 
related to relevant criteria, for example in elemental cost break- 
downs . 

3.3 Using indicators in evaluation 

'Norm' values of such indicators as derived from a databank 
could convey the industry standard under average circumstances. 
These can be weighted by factors (based on 'weighting indicators') 
to adjust for special project conditions and client priorities. 
Evaluators would then compare actual performance against such 
weighted 'norms' . 
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KEY NOTE 

Targetted Profile The further a point is from the origin 

Achieved Profile along any criterion (or sub-criterion) 

axis: the better is the performance 
against that criterion (or sub-criterion). 


EXAMPLE OF A PROJECT PERFORMANCE PROFILE 




FIGURE 1 



Figure 2 illustrates how deviations from the norms, say in 
certain primary indicators, would alert the evaluator to 
investigate relevant areas further; by checking out the secondary 
(and tertiary) indicators therein. Thus the evaluator is directed 
through a structured search for the root causes of under- 
performance"^ . 

However two caveats are noted. Firstly apparently 'normal' 
primary indicators may mask a 'delinquent' secondary indicator 
which is over -compensated by over-performance against a parallel 
secondary indicator. Secondly parallel qualitative assessments are 
essential to place apparent 'good' or 'bad' variances in context. 

3.4 Examples from international mega-project ev aluation 

International funding agencies such as USAID, the overseas 
Development Administration (ODA) and other bilateral aid agencies 
have used indicators - based evaluation systems to measure the 
success levels of their programs in other countries. Figures 3 and 
4 illustrate the evaluation frameworks used for such systems. 
Different cultures, diverse value systems, divergent views and 
conflicting claims would have proved the worth of such a 
systematized approach to evaluation in such situations. Similar 
scenarios would arise in the multinational mega- projects of the 
present and future . 

4. PROBLEMS AND PROSPECTS OF MEGA-PROJECT EVALUATION 
4.1 The changing nature of mega -pro j ects 

While projects were always about change, the mega- projects 
of today reflect the ultra high rates of change that technology 
has facilitated. The multiplicity of participants and the effects 
of the project increasingly transcend national boundaries; as 
in joint venture in space exploration and environmental control. 
Technology advances much faster than the frameworks needed to 
manage it, generating human and environmental stresses and 
strains . 

Since mega - pro j ec ts may take longer than the average 
projects, the success criteria are more susceptible to revision 
eg. with the change of governments or other project stake -holders . 
Multi-attribute evaluation becomes more complex with such shifting 
goal-posts. Even on a macro scale Purchasing Power Parity and 
various forms of Quality of Life indices are supplanting GNP . 
Environmental impact indicators are another growth industry. A 
mega-project evaluator needs to quantify social costs and benefits 
as well . 

The author found specialist software designed for 
Integrating a series of pairwise comparisons to be useful in 
reducing the residual subjectivity of some such basically 
qualitative assessments . It is also useful to make allowances for 
possible distortions by effects such as the 'Halo' effect, th^ 
'Hawthorne' effect, the 'Placebo' effect or the 'Harvard fallacy' 7 
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The matrix or hybrid project matrix structures typical of 
projects may be inadequate to convey the mechanisms of mega- 
project operations. Additional dimensions are introduced by the 
multiple stake-holders . 

For example the consortium of contractors on the Channel 
tunnel is composed of 5 major British and 5 major French 
contractors. The Eurotunnel client is similarly diverse. Joint 
ventures in Aerospace have exposed project participants to even 
wider ranges of diversity. Build, Operate and Transfer (BOT) type 
contracts (and their variations) for major infrastructure 
projects bring together an array of stake-holders with an apparent 
multiplicity of objectives, from various countries and cultures. 

Thus a third dimension (at the very least) is needed in the 
organisational picture to accommodate the multiple organisations, 
each of which have their own functional hierarchies; from each of 
which in turn, project participants will be drawn. The 
participants therefore have linkages (however weak) to their 
parent organisations as well as to their functional disciplines. 
The nature of each industry (and specific factors such as mobility 
therein) would determine the relative strengths of such linkages. 
Evaluators of mega-projects need to make allowances for such 
linkages and loyalties; and to be aware of the particular 
management structures, styles and operative information flow 
mechanisms . 

Int e grating evaluation mechanisms into management systems 

Another characteristic of most mega-projects is that the 
special priorities and conditions governing them are that much 
more different from the average. While every project is unique, 
and therefore diverges from each other in most respects, such 
divergences are magnified many fold in mega-projects. However the 
assimilation and analysis of databanks of relevant information 
from ' similar ' projects is an important step in placing the 
project in context. 

Project planning is another critical function in such mega- 
projects. The multi-attribute success criteria of multiple stake- 
holders is hopefully translated into meaningful management targets. 

It is an obvious advantage to build in monitoring, evaluation 
and control (remedial) systems to integrate with the management 
systems. For instance the management information system 
disseminating information required to run the job can be designed 
to retrieve reports for review. The Quality Assurance or Safety 
systems could have similar built In evaluation mechanisms. 

'Bechtel' which manages many mega-projects has installed 
systems where the 'Controls' function incorporating progress and 
cost control personnel, is separated from the 'Management' 
function; perhaps so that an independent assessment is 
facilitated. However the actual information gathering could be 
integrated to prevent duplication; apart from specific 
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investigations from time to time which are necessary to vary the 
procedures so as to minimise cheating; to cater to specific 
situations and to evaluate from fresh angles. 

4.3 Integrating progress, cost and quality m anagement systems 

Progress and cost control systems have been advantageously 
Integrated under 'earned value ' analysis where cost and schedule 
variances are generated. The U.S. Departments of Defence and 
Energy for example have used ' cost/ schedule control system 
criteria' (C/SCSC) to interrelate work scope definition, sche^le, 
estimate/ budget, physical performance and actual expenditure 

'Cost trending' is a similar procedure used for example, In 
the Airport Core Programme in Hong Kong where project cost, scope 
and programming are continuously monitored and hopefully 
controlled . 

Recent international emphasis on quality, 'galvanised' 
contractors, consultants and even large client organisations to 
seek accredition under ISO 9000 or the relevant local standards 
such as BS5750 . Quality now has to be 'designed, in' 'built in' 
and 'assured' rather than checked or inspected for after the 
event. One of the key advantages of installing internal systems 
geared to such accredition is the opportunity to incorporate 
recording and reporting sub -systems that can also service the 
monitoring and evaluation functions. The discipline instilled in 
maintaining these provisions, helps overcome previous resistance 
to such procedures. 

Another relatively recent concept is that of using 
'feedforward' mechanisms that alert management to deviations in 
inputs; rather than relying only on feedback from output 
variances . 

4.4 Trends in mega-proi ects ; and related cr iteria and indicators 

of evaluation 

Jargon such as ' out - sourcing' ' down- loading ' and ' down- 
sizing' operations, convey the moves towards sub-contracting out 
specific activities to specialists while whittling down one's own 
operations to core activities and interface management. 'Joint 
venturing' and 'partnering' are other manifestations of similar 
trends. Claims of interface interference by the many interacting 
parties can thus prove crucial. 

This leads to criteria and indicators such as numbers and 
values of claims, related to numbers of such interfaces and values 
of such work packages. 

Some clients and consultants even use (official or 
unofficial) indicators of the 'claims - consciousness' of 
contractors, in order to weight tenders accordingly. Of course 
such indicators should themselves be weighted by the circumstances 
(of justification or otherwise) of the original claims. Diverse 



sources of information contribute to construct such indicators, 
eg: the number of referrals to arbitration; the number of 
rejections therefrom; comparisons with other contractors. 

Disputes themselves are not settled only by litigation or 
arbitration but by mediation, conciliation etc. Alternative 
dispute Resolution (ADR) underlines the philosophy of such new 
approaches; just as alternative procurement systems are spawning 
many variations of contracting for design, supply and construction 
services. For instance alternative contracting is now challenging 
the traditional lowest - bid system even in public - sector 
construction in USA 1 . Even payments are linked to the statistical 
quality of work, for example on some New Jersey DOT sponsored 
highway construction projects. Statistical indicators of concrete 
or asphalt strength, thickness, smoothness and riding quality can 
enhance contractors payments up to 103% of contract value. The 
trade-off is in less maintenance and future repairs. 

Impact evaluation would continue to rely heavily on variance 
indicators to assess deviations from set targets, while 
implementation evaluation also uses indicators of resource 
utilisation or resource idling rates. The evaluation of social and 
environmental costs and benefits in computing Economic Rates of 
Return for example, are areas where evaluation expertise needs 
refinement . 

The effectiveness of technology transfer is a sensitive area 
in mega-projects that straddle national boundaries. The evaluation 
of cross -benefi ts that accrue to, and costs incurred by joint- 
venture partners is another crucial area where criteria and 
indicators used are often inadequate to track the longer term 
impacts and trace the wider repercussions of participation in 
such mega-projects. The Asia & Pacific Centre for Transfer of 
Technology (of ESCAP) formulated a system of evaluating technology 
content using sets of indicators related to 4 different aspects of 
technology 

Trends appear to favour the increasing 'size' of mega- 
projects; as the benefits of comparative advantages (for instance 
of different operations in different nations) and of synergistic 
linkages gradually overcome traditional apprehensions. 'Drivers' 
are also derived from providing opportunities to regions in 
temporary recession or decline for example; while resistance 
arises for instance from heightened environmental apprehensions 
aroused by mega -impact projects. 

However the need to tackle environmental, Space exploration 
and even ocean exploitation endeavours on a broader basis, justify 
the overall push towards globalisation. This will perhaps herald 
the supra-mega-projects of the future. 

Such trends highlight the needs not only to develop 
technologies of supra-mega -project management, but also for their 
realistic evaluation, so as to continue improving that management. 



4.5 Concluding observations 


'Trade- offs' are often warranted where for example high 
performance against one criterion is sacrificed to enable 
exceptional performance against others which are of higher 
priority, as for example illustrated in Figure 5. While such 
'trade-offs 1 are easier appreciated and accommodated on small 
projects with less stake-holders, they become the source of 
conflict in mul t i - par t ic ipant me ga - pro j ec t s . Even if agreed, 
unless it is explicity so, these could lead to conflicts and 
unfair future evaluations . The moral is the enhanced importance 
of a detailed project brief in mega-projects; and also the 
criticality of the conceptualisation phase where such trade - offs 
are best incorporated; rather than making adjustments mid- stream 
in the project when the costs of disruption can be tremendous. 

The multi- dimensional character of mega-projects arise 
from the usually multi-disciplinary, mul t i - cul tural multiple 
stake -holders and warrant multi - attribute evaluations which can 
become somewhat complex, and more so in view of the foregoing 
trade-offs, that may continue in the dynamic project environment. 

The purpose of this paper is to focus on the evaluation of 
mega-projects and to highlight the need for a systematisation of 
approaches based on multi - dimensional criteria (and sub- criteria) 
and appropriate indicators for measuring against such criteria 
and sub -criteria. Data banks of previous project parameters are 
useful to assess and weight 'norm values' of such indicators so as 
to facilitate more realistic expectations. Their usefulness 
presupposes classification according to appropriate categories and 
sub-categories. The multiplicity of such variables (criteria, 
indicators and weighting factors) and the ranges of possible 
values led the author to propose (a) a modular structure and (b) a 
knowledge -based front-end to facilitate the efficient use of such 
a system. It's viability was demonstrated in a pilot construction 
project evaluation system . 

The evaluation system should not grow so complex as to 'hide 
the wood for the trees'. Furthermore the rewards from evaluation 
as in improved performance on on-going and future projects should 
exceed the costs of the evaluation itself. Such objectives are 
easier to ensure through the foregoing systematisation of broad 
approaches to evaluation. A given broad core approach can be 
selected to suit a particular category of projects ; and then 
modified according to particular project priorities and contextual 
conditions^ . 

The integration of evaluation mechanisms into the planning, 
operational and information sub-systems further facilitates 
efficient evaluations. The next question that surfaces relates to 
the effectiveness and efficiency of the evaluation itself; 
possibly leading on to the evaluation of the evaluations and 
evaluators themselves. 
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